You are currently looking at version 1.1 of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the Jupyter Notebook FAQ course resource.



In [3]:
import pandas as pd
import numpy as np
from scipy.stats import ttest_ind
import re

Assignment 4 - Hypothesis Testing

This assignment requires more individual learning than previous assignments - you are encouraged to check out the pandas documentation to find functions or methods you might not have used yet, or ask questions on Stack Overflow and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff.

Definitions:

  • A quarter is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.
  • A recession is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.
  • A recession bottom is the quarter within a recession which had the lowest GDP.
  • A university town is a city which has a high percentage of university students compared to the total population of the city.

Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (price_ratio=quarter_before_recession/recession_bottom)

The following data files are available for this assignment:

  • From the Zillow research data site there is housing data for the United States. In particular the datafile for all homes at a city level, City_Zhvi_AllHomes.csv, has median home sale prices at a fine grained level.
  • From the Wikipedia page on college towns is a list of university towns in the United States which has been copy and pasted into the file university_towns.txt.
  • From Bureau of Economic Analysis, US Department of Commerce, the GDP over time of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file gdplev.xls. For this assignment, only look at GDP data from the first quarter of 2000 onward.

Each function in this assignment below is worth 10%, with the exception of run_ttest(), which is worth 50%.


In [212]:
# Use this dictionary to map state names to two letter acronyms
states = {'OH': 'Ohio', 'KY': 'Kentucky', 'AS': 'American Samoa', 'NV': 'Nevada', 'WY': 'Wyoming', 'NA': 'National', 'AL': 'Alabama', 'MD': 'Maryland', 'AK': 'Alaska', 'UT': 'Utah', 'OR': 'Oregon', 'MT': 'Montana', 'IL': 'Illinois', 'TN': 'Tennessee', 'DC': 'District of Columbia', 'VT': 'Vermont', 'ID': 'Idaho', 'AR': 'Arkansas', 'ME': 'Maine', 'WA': 'Washington', 'HI': 'Hawaii', 'WI': 'Wisconsin', 'MI': 'Michigan', 'IN': 'Indiana', 'NJ': 'New Jersey', 'AZ': 'Arizona', 'GU': 'Guam', 'MS': 'Mississippi', 'PR': 'Puerto Rico', 'NC': 'North Carolina', 'TX': 'Texas', 'SD': 'South Dakota', 'MP': 'Northern Mariana Islands', 'IA': 'Iowa', 'MO': 'Missouri', 'CT': 'Connecticut', 'WV': 'West Virginia', 'SC': 'South Carolina', 'LA': 'Louisiana', 'KS': 'Kansas', 'NY': 'New York', 'NE': 'Nebraska', 'OK': 'Oklahoma', 'FL': 'Florida', 'CA': 'California', 'CO': 'Colorado', 'PA': 'Pennsylvania', 'DE': 'Delaware', 'NM': 'New Mexico', 'RI': 'Rhode Island', 'MN': 'Minnesota', 'VI': 'Virgin Islands', 'NH': 'New Hampshire', 'MA': 'Massachusetts', 'GA': 'Georgia', 'ND': 'North Dakota', 'VA': 'Virginia'}

In [229]:
def get_list_of_university_towns():
    '''Returns a DataFrame of towns and the states they are in from the 
    university_towns.txt list. The format of the DataFrame should be:
    DataFrame( [ ["Michigan", "Ann Arbor"], ["Michigan", "Yipsilanti"] ], 
    columns=["State", "RegionName"]  )
    
    The following cleaning needs to be done:

    1. For "State", removing characters from "[" to the end.
    2. For "RegionName", when applicable, removing every character from " (" to the end.
    3. Depending on how you read the data, you may need to remove newline character '\n'. '''
    
    df = pd.read_csv("university_towns.txt", delimiter="\n", header=None)
    ans = df.head()
    
    # clean up
    #df['state'] = df[df[0]  "[edit]"]
    
    def func(row):
        if row[0][-6:] == '[edit]':
            return row[0][:-6]
        else:
            return False
        
    #df['state'] = df.apply(func, axis=1)
    df['State'] = df.loc[df[0].str.contains('[edit]', regex=False), 0]
    df['State'] = df['State'].str.replace('\[edit]', "")
    
    df['RegionName'] = df.loc[df['State'].isnull(), 0]
    df['RegionName'] = df['RegionName'].str.replace('\([^)]*\)', '')
    df['RegionName'] = df['RegionName'].str.replace('[0-9]+', '')
    df['RegionName'] = df['RegionName'].str.replace('\[\]', '')
    
    df['State'] =  df['State'].ffill()
    df = df.dropna()
    df['RegionName'] = df['RegionName'].str.rstrip()
    
    cols = ['State', 'RegionName']
    df = df[cols].reset_index(drop=True)
    
    return df


Out[229]:
State RegionName
0 Alabama Auburn
1 Alabama Florence
2 Alabama Jacksonville
3 Alabama Livingston
4 Alabama Montevallo
5 Alabama Troy
6 Alabama Tuscaloosa
7 Alabama Tuskegee
8 Alaska Fairbanks
9 Arizona Flagstaff
10 Arizona Tempe
11 Arizona Tucson
12 Arkansas Arkadelphia
13 Arkansas Conway
14 Arkansas Fayetteville
15 Arkansas Jonesboro
16 Arkansas Magnolia
17 Arkansas Monticello
18 Arkansas Russellville
19 Arkansas Searcy
20 California Angwin
21 California Arcata
22 California Berkeley
23 California Chico
24 California Claremont
25 California Cotati
26 California Davis
27 California Irvine
28 California Isla Vista
29 California University Park, Los Angeles
... ... ...
487 Virginia Wise
488 Virginia Chesapeake
489 Washington Bellingham
490 Washington Cheney
491 Washington Ellensburg
492 Washington Pullman
493 Washington University District, Seattle
494 West Virginia Athens
495 West Virginia Buckhannon
496 West Virginia Fairmont
497 West Virginia Glenville
498 West Virginia Huntington
499 West Virginia Montgomery
500 West Virginia Morgantown
501 West Virginia Shepherdstown
502 West Virginia West Liberty
503 Wisconsin Appleton
504 Wisconsin Eau Claire
505 Wisconsin Green Bay
506 Wisconsin La Crosse
507 Wisconsin Madison
508 Wisconsin Menomonie
509 Wisconsin Milwaukee
510 Wisconsin Oshkosh
511 Wisconsin Platteville
512 Wisconsin River Falls
513 Wisconsin Stevens Point
514 Wisconsin Waukesha
515 Wisconsin Whitewater
516 Wyoming Laramie

517 rows × 2 columns


In [232]:
def get_recession_start():
    '''Returns the year and quarter of the recession start time as a 
    string value in a format such as 2005q3'''
    df = pd.read_excel('gdplev.xls')
    date = df.columns[7]
    df.drop(df.columns[[3,7]], axis=1, inplace=True)
    
    df = df[df.columns[-3:]]
    col_names = ['Quarter', df.iloc[4,1], df.iloc[4,2]]
    
# #     col_names = [df.iloc[1,0], df.iloc[4,1], df.iloc[4,2], df.iloc[2,3],
# #                  df.iloc[4,4], df.iloc[4,5]]
    df.columns = col_names # column names
    df = df[7:].reset_index(drop=True) # remove NaN
    loc = df[df['Quarter'] == '2000q1'].index.tolist()[0]
    
    df = df[loc:].reset_index(drop=True)
    
    quarters = []
    for i in range(len(df) - 2):
        if ((df.iloc[i, 1] > df.iloc[i+1, 1]) & (df.iloc[i+1, 1] > df.iloc[i+2,1])):
            quarters.append(df.iloc[i,0])
    
    return quarters[0]


Out[232]:
'2008q3'

In [233]:
def get_recession_end():
    '''Returns the year and quarter of the recession end time as a 
    string value in a format such as 2005q3'''
    
    df = pd.read_excel('gdplev.xls')
    df.drop(df.columns[[3,7]], axis=1, inplace=True)
    
    df = df[df.columns[-3:]]
    col_names = ['Quarter', df.iloc[4,1], df.iloc[4,2]]
    
    df.columns = col_names
    df = df[7:].reset_index(drop=True)
    
    start = get_recession_start()
    start_idx = df[df['Quarter'] == start].index.tolist()[0]
    df = df.loc[start_idx:, :]
    
    quarters = []
    for row in range(len(df) - 2):
        if (df.iloc[row, 1] < df.iloc[row+1, 1]) & (df.iloc[row+1,1] < df.iloc[row+2, 1]):
            return df.iloc[row+2, 0] #should be just row i think
    return


Out[233]:
'2009q4'

In [234]:
def get_recession_bottom():
    '''Returns the year and quarter of the recession bottom time as a 
    string value in a format such as 2005q3'''
    
    df= pd.read_excel('gdplev.xls')
    df.drop(df.columns[[3,7]], axis=1, inplace=True)
    
    df = df[df.columns[-3:]]
    col_names = ['Quarter', df.iloc[4,1], df.iloc[4,2]]
    df.columns = col_names
    
    df = df[7:].reset_index(drop=True)
    
    start = get_recession_start()
    start_idx = df[df['Quarter'] == start].index.tolist()[0]
    end = get_recession_end()
    end_idx = df[df['Quarter'] == end].index.tolist()[0]
    df = df[start_idx:end_idx]
    bottom = df[df.columns[1]].idxmin()
    
    return df.loc[bottom, df.columns[0]]


Out[234]:
'2009q2'

In [236]:
def convert_housing_data_to_quarters():
    '''Converts the housing data to quarters and returns it as mean 
    values in a dataframe. This dataframe should be a dataframe with
    columns for 2000q1 through 2016q3, and should have a multi-index
    in the shape of ["State","RegionName"].
    
    Note: Quarters are defined in the assignment description, they are
    not arbitrary three month periods.
    
    The resulting dataframe should have 67 columns, and 10,730 rows.
    '''
    df = pd.read_csv('City_Zhvi_AllHomes.csv')
    
    def new_col_names():
        quarters = ['q1', 'q2', 'q3', 'q4']
        yr_qr = []
        for year in range(2000, 2017):
            for q in quarters:
                yr_qr.append(str(year)+q)
        return yr_qr[:67]
    
    # clean up data, removing unnecessary columns
    df.drop(df.columns[[0,3,4,5]], axis=1, inplace=1)
    df['State'].replace(states, inplace=1)
    df.set_index(['State', 'RegionName'], inplace=1)
    drop_cols = df.columns[df.columns.str.startswith('199')]
    df.drop(drop_cols, axis=1, inplace=1)
    
    
    # get the quarters from 
    qr = []
    for i in range(0, len(df.columns),3):
        q = list(df.columns[i:i+3])
        qr.append(q)
        
    col_names = new_col_names()
    for col, qr in zip(col_names, qr):
        df[col] = df[qr].mean(axis=1)
    
    return df[col_names]
convert_housing_data_to_quarters()


Out[236]:
2000q1 2000q2 2000q3 2000q4 2001q1 2001q2 2001q3 2001q4 2002q1 2002q2 ... 2014q2 2014q3 2014q4 2015q1 2015q2 2015q3 2015q4 2016q1 2016q2 2016q3
State RegionName
New York New York NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 5.151000e+05 5.189667e+05 5.224333e+05 5.237333e+05 5.320667e+05 5.484333e+05 5.620667e+05 5.724333e+05 5.838000e+05 6.025667e+05
California Los Angeles 2.070667e+05 2.144667e+05 2.209667e+05 2.261667e+05 2.330000e+05 2.391000e+05 2.450667e+05 2.530333e+05 2.619667e+05 2.727000e+05 ... 4.982667e+05 5.093000e+05 5.179667e+05 5.266667e+05 5.349667e+05 5.453333e+05 5.571000e+05 5.679667e+05 5.805667e+05 5.900667e+05
Illinois Chicago 1.384000e+05 1.436333e+05 1.478667e+05 1.521333e+05 1.569333e+05 1.618000e+05 1.664000e+05 1.704333e+05 1.755000e+05 1.775667e+05 ... 1.881333e+05 1.902333e+05 1.937333e+05 1.948000e+05 1.989333e+05 2.016333e+05 2.028667e+05 2.037333e+05 2.056000e+05 2.080000e+05
Pennsylvania Philadelphia 5.300000e+04 5.363333e+04 5.413333e+04 5.470000e+04 5.533333e+04 5.553333e+04 5.626667e+04 5.753333e+04 5.913333e+04 6.073333e+04 ... 1.145667e+05 1.158667e+05 1.166000e+05 1.167333e+05 1.179333e+05 1.207000e+05 1.221667e+05 1.237000e+05 1.260000e+05 1.305667e+05
Arizona Phoenix 1.118333e+05 1.143667e+05 1.160000e+05 1.174000e+05 1.196000e+05 1.215667e+05 1.227000e+05 1.243000e+05 1.265333e+05 1.283667e+05 ... 1.663000e+05 1.675333e+05 1.705000e+05 1.732667e+05 1.764667e+05 1.806000e+05 1.851333e+05 1.889000e+05 1.931000e+05 1.986333e+05
Nevada Las Vegas 1.326000e+05 1.343667e+05 1.354000e+05 1.370000e+05 1.395333e+05 1.417333e+05 1.433667e+05 1.461333e+05 1.493333e+05 1.509333e+05 ... 1.689667e+05 1.715667e+05 1.746333e+05 1.776000e+05 1.802000e+05 1.851667e+05 1.887667e+05 1.922333e+05 1.960333e+05 2.026667e+05
California San Diego 2.229000e+05 2.343667e+05 2.454333e+05 2.560333e+05 2.672000e+05 2.762667e+05 2.845000e+05 2.919333e+05 3.012333e+05 3.128667e+05 ... 4.745667e+05 4.813000e+05 4.862333e+05 4.934333e+05 5.032333e+05 5.129667e+05 5.210000e+05 5.276333e+05 5.318333e+05 5.431333e+05
Texas Dallas 8.446667e+04 8.386667e+04 8.486667e+04 8.783333e+04 8.973333e+04 8.930000e+04 8.906667e+04 9.090000e+04 9.256667e+04 9.380000e+04 ... 1.053000e+05 1.071667e+05 1.099333e+05 1.124667e+05 1.199667e+05 1.280667e+05 1.345333e+05 1.398000e+05 1.453000e+05 1.554333e+05
California San Jose 3.742667e+05 4.065667e+05 4.318667e+05 4.555000e+05 4.706667e+05 4.702000e+05 4.568000e+05 4.455667e+05 4.414333e+05 4.577667e+05 ... 6.814333e+05 7.015333e+05 7.186667e+05 7.373333e+05 7.631333e+05 7.884333e+05 8.062333e+05 8.244667e+05 8.385667e+05 8.364667e+05
Florida Jacksonville 8.860000e+04 8.970000e+04 9.170000e+04 9.310000e+04 9.440000e+04 9.560000e+04 9.706667e+04 9.906667e+04 1.012333e+05 1.034333e+05 ... 1.189333e+05 1.193000e+05 1.204667e+05 1.221667e+05 1.250667e+05 1.292000e+05 1.310667e+05 1.335667e+05 1.373667e+05 1.398000e+05
California San Francisco 4.305000e+05 4.644667e+05 4.835333e+05 4.930000e+05 4.940667e+05 4.961333e+05 5.041000e+05 5.134000e+05 5.204333e+05 5.381667e+05 ... 9.261333e+05 9.546333e+05 9.743000e+05 1.005200e+06 1.063367e+06 1.105067e+06 1.121167e+06 1.146500e+06 1.132833e+06 1.121367e+06
Texas Austin 1.429667e+05 1.452667e+05 1.494667e+05 1.557333e+05 1.612333e+05 1.607333e+05 1.595333e+05 1.600333e+05 1.589667e+05 1.575000e+05 ... 2.473000e+05 2.516333e+05 2.573667e+05 2.643000e+05 2.729000e+05 2.807000e+05 2.868667e+05 2.929000e+05 2.992667e+05 3.016333e+05
Michigan Detroit 6.620000e+04 6.830000e+04 6.676667e+04 6.703333e+04 6.753333e+04 6.836667e+04 6.926667e+04 6.996667e+04 7.100000e+04 7.233333e+04 ... 3.523333e+04 3.510000e+04 3.536667e+04 3.440000e+04 3.336667e+04 3.410000e+04 3.416667e+04 3.406667e+04 3.420000e+04 3.596667e+04
Ohio Columbus 9.436667e+04 9.583333e+04 9.713333e+04 9.826667e+04 9.940000e+04 1.002667e+05 1.010667e+05 1.022000e+05 1.034000e+05 1.048000e+05 ... 1.041000e+05 1.057000e+05 1.068333e+05 1.081333e+05 1.106667e+05 1.127000e+05 1.153000e+05 1.160000e+05 1.172000e+05 1.202667e+05
Tennessee Memphis 7.250000e+04 7.320000e+04 7.386667e+04 7.400000e+04 7.416667e+04 7.493333e+04 7.550000e+04 7.606667e+04 7.633333e+04 7.676667e+04 ... 6.920000e+04 7.053333e+04 7.206667e+04 7.083333e+04 6.870000e+04 6.896667e+04 7.003333e+04 7.103333e+04 7.416667e+04 7.696667e+04
North Carolina Charlotte 1.269333e+05 1.283667e+05 1.302000e+05 1.315667e+05 1.329333e+05 1.332000e+05 1.328000e+05 1.331000e+05 1.343667e+05 1.353667e+05 ... 1.495667e+05 1.502000e+05 1.526000e+05 1.551000e+05 1.590333e+05 1.613333e+05 1.635333e+05 1.670333e+05 1.695000e+05 1.732000e+05
Texas El Paso 7.626667e+04 7.686667e+04 7.673333e+04 7.730000e+04 7.823333e+04 7.830000e+04 7.743333e+04 7.680000e+04 7.660000e+04 7.640000e+04 ... 1.115333e+05 1.115333e+05 1.115000e+05 1.114000e+05 1.116000e+05 1.115333e+05 1.109667e+05 1.113667e+05 1.121333e+05 1.127667e+05
Massachusetts Boston 2.069333e+05 2.191667e+05 2.331000e+05 2.425000e+05 2.496000e+05 2.570667e+05 2.669333e+05 2.749667e+05 2.825000e+05 2.893000e+05 ... 4.223667e+05 4.299667e+05 4.380667e+05 4.485000e+05 4.542000e+05 4.646000e+05 4.782000e+05 4.912333e+05 4.989667e+05 5.103000e+05
Washington Seattle 2.486000e+05 2.556000e+05 2.625333e+05 2.674000e+05 2.710000e+05 2.724333e+05 2.741333e+05 2.781667e+05 2.805000e+05 2.846000e+05 ... 4.415333e+05 4.514667e+05 4.588667e+05 4.710667e+05 4.976667e+05 5.191333e+05 5.407000e+05 5.616667e+05 5.856000e+05 6.055000e+05
Maryland Baltimore NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.109333e+05 1.110667e+05 1.096000e+05 1.100333e+05 1.130000e+05 1.154333e+05 1.161333e+05 1.164333e+05 1.177000e+05 1.189000e+05
Colorado Denver 1.622333e+05 1.678333e+05 1.743333e+05 1.803333e+05 1.865000e+05 1.925333e+05 1.964000e+05 1.991000e+05 2.012333e+05 2.024333e+05 ... 2.651333e+05 2.750333e+05 2.866667e+05 2.978000e+05 3.130667e+05 3.246667e+05 3.336667e+05 3.404000e+05 3.472333e+05 3.561667e+05
District of Columbia Washington 1.377667e+05 1.442000e+05 1.487000e+05 1.477000e+05 1.497667e+05 1.551333e+05 1.646333e+05 1.725333e+05 1.804667e+05 1.933000e+05 ... 4.477333e+05 4.565667e+05 4.624667e+05 4.675000e+05 4.878667e+05 5.014333e+05 5.064667e+05 5.061333e+05 5.105333e+05 5.228000e+05
Tennessee Nashville 1.138333e+05 1.152667e+05 1.158667e+05 1.169333e+05 1.180333e+05 1.191667e+05 1.201000e+05 1.208000e+05 1.215667e+05 1.226333e+05 ... 1.613333e+05 1.635333e+05 1.677667e+05 1.724667e+05 1.760000e+05 1.820667e+05 1.895667e+05 1.953333e+05 2.001333e+05 2.072333e+05
Wisconsin Milwaukee 7.803333e+04 7.906667e+04 8.103333e+04 8.233333e+04 8.403333e+04 8.556667e+04 8.706667e+04 8.840000e+04 8.953333e+04 9.136667e+04 ... 9.206667e+04 9.260000e+04 9.263333e+04 9.496667e+04 9.533333e+04 9.553333e+04 9.576667e+04 9.590000e+04 9.746667e+04 1.003333e+05
Arizona Tucson 1.018333e+05 1.029667e+05 1.044667e+05 1.056667e+05 1.072000e+05 1.087667e+05 1.105667e+05 1.128000e+05 1.150000e+05 1.172000e+05 ... 1.423333e+05 1.429333e+05 1.442667e+05 1.442000e+05 1.446667e+05 1.457667e+05 1.476000e+05 1.495333e+05 1.517000e+05 1.537000e+05
Oregon Portland 1.528000e+05 1.547667e+05 1.565667e+05 1.574667e+05 1.599000e+05 1.618000e+05 1.642667e+05 1.677667e+05 1.707667e+05 1.741000e+05 ... 2.831000e+05 2.889000e+05 2.971667e+05 3.047667e+05 3.159333e+05 3.337000e+05 3.527333e+05 3.709667e+05 3.862000e+05 3.988000e+05
Oklahoma Oklahoma City 7.643333e+04 7.750000e+04 7.856667e+04 7.916667e+04 7.983333e+04 8.040000e+04 8.113333e+04 8.173333e+04 8.260000e+04 8.343333e+04 ... 1.151000e+05 1.153667e+05 1.160667e+05 1.173000e+05 1.210333e+05 1.237000e+05 1.258667e+05 1.267000e+05 1.284333e+05 1.295667e+05
Nebraska Omaha 1.128000e+05 1.141000e+05 1.167333e+05 1.189000e+05 1.208667e+05 1.197667e+05 1.178667e+05 1.174000e+05 1.180667e+05 1.176333e+05 ... 1.301667e+05 1.305333e+05 1.326000e+05 1.330667e+05 1.352667e+05 1.373667e+05 1.398000e+05 1.411667e+05 1.424667e+05 1.440667e+05
New Mexico Albuquerque 1.258667e+05 1.267000e+05 1.264333e+05 1.267333e+05 1.271000e+05 1.277333e+05 1.285667e+05 1.299000e+05 1.310667e+05 1.321000e+05 ... 1.623667e+05 1.632333e+05 1.642000e+05 1.659000e+05 1.663333e+05 1.692667e+05 1.691000e+05 1.700333e+05 1.710333e+05 1.731667e+05
California Fresno 9.410000e+04 9.526667e+04 9.646667e+04 9.823333e+04 1.005667e+05 1.035667e+05 1.072333e+05 1.103000e+05 1.140333e+05 1.185333e+05 ... 1.713000e+05 1.748667e+05 1.789000e+05 1.804667e+05 1.823667e+05 1.867000e+05 1.895333e+05 1.908000e+05 1.942333e+05 1.979333e+05
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Texas Granite Shoals NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.118333e+05 1.112667e+05 1.113000e+05 1.127000e+05 1.176000e+05 1.218667e+05 1.256667e+05 1.278333e+05 1.289000e+05 1.302667e+05
Maryland Piney Point 1.556667e+05 1.551667e+05 1.584667e+05 1.637000e+05 1.634000e+05 1.648333e+05 1.647000e+05 1.679000e+05 1.782667e+05 1.812000e+05 ... 2.828333e+05 2.922333e+05 2.829667e+05 2.727000e+05 2.767333e+05 2.763667e+05 2.756000e+05 2.830333e+05 2.823667e+05 2.844000e+05
California Stinson Beach 9.974333e+05 1.120867e+06 1.247000e+06 1.363000e+06 1.424133e+06 1.468167e+06 1.375567e+06 1.275233e+06 1.244367e+06 1.145100e+06 ... 2.243067e+06 2.432533e+06 2.587300e+06 2.651133e+06 2.692367e+06 2.606400e+06 2.729900e+06 2.955300e+06 2.949600e+06 2.894600e+06
Wisconsin Maribel NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.280333e+05 1.280667e+05 1.286333e+05 1.299333e+05 1.291333e+05 1.270333e+05 1.245667e+05 1.230667e+05 1.257333e+05 1.327333e+05
Idaho Middleton 1.060667e+05 1.043333e+05 1.019000e+05 1.041667e+05 1.061667e+05 1.083667e+05 1.110333e+05 1.112333e+05 1.141000e+05 1.141667e+05 ... 1.434667e+05 1.463667e+05 1.480000e+05 1.492000e+05 1.498333e+05 1.526000e+05 1.551667e+05 1.565333e+05 1.613333e+05 1.655000e+05
Kentucky Almo 8.403333e+04 8.526667e+04 8.543333e+04 8.556667e+04 8.710000e+04 8.843333e+04 8.833333e+04 8.753333e+04 8.743333e+04 9.123333e+04 ... 1.211667e+05 1.243667e+05 1.243000e+05 1.239333e+05 1.248000e+05 1.251667e+05 1.247333e+05 1.245000e+05 1.280667e+05 1.282000e+05
Colorado Bennett 1.329000e+05 1.358333e+05 1.398000e+05 1.446667e+05 1.483000e+05 1.521000e+05 1.542333e+05 1.562000e+05 1.587333e+05 1.606333e+05 ... 1.508000e+05 1.595000e+05 1.681667e+05 1.775667e+05 1.869000e+05 1.925000e+05 1.953333e+05 2.002333e+05 2.077667e+05 2.171667e+05
New Hampshire East Hampstead 1.618000e+05 1.691000e+05 1.739667e+05 1.805000e+05 1.909000e+05 1.950667e+05 1.992667e+05 2.074000e+05 2.123000e+05 2.122333e+05 ... 2.525667e+05 2.536000e+05 2.552333e+05 2.582000e+05 2.619000e+05 2.623667e+05 2.676333e+05 2.756000e+05 2.792000e+05 2.831333e+05
Missouri Garden City NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.051000e+05 1.047667e+05 1.067667e+05 1.092333e+05 1.046667e+05 1.043333e+05 1.078667e+05 1.093667e+05 1.143667e+05 1.158000e+05
Arkansas Mountainburg 5.716667e+04 6.433333e+04 6.783333e+04 6.900000e+04 6.866667e+04 6.386667e+04 6.376667e+04 6.546667e+04 6.533333e+04 6.600000e+04 ... 8.043333e+04 8.153333e+04 8.376667e+04 8.426667e+04 7.973333e+04 7.773333e+04 8.040000e+04 8.396667e+04 8.966667e+04 9.586667e+04
Tennessee Ocoee 5.693333e+04 5.313333e+04 5.620000e+04 6.113333e+04 6.160000e+04 6.543333e+04 6.710000e+04 6.763333e+04 6.513333e+04 6.770000e+04 ... 1.301333e+05 1.332333e+05 1.327000e+05 1.315000e+05 1.282333e+05 1.263000e+05 1.272667e+05 1.305667e+05 1.346333e+05 1.400667e+05
New York Upper Brookville 1.230967e+06 1.230967e+06 1.237700e+06 1.261567e+06 1.295167e+06 1.340033e+06 1.403667e+06 1.481967e+06 1.536167e+06 1.562033e+06 ... 1.715767e+06 1.723433e+06 1.761667e+06 1.793633e+06 1.837833e+06 1.803100e+06 1.807200e+06 1.863933e+06 1.904933e+06 1.978233e+06
Hawaii Volcano 9.870000e+04 1.053667e+05 1.146667e+05 1.247667e+05 1.181333e+05 1.194000e+05 1.232667e+05 1.211667e+05 1.233000e+05 1.169000e+05 ... 1.910667e+05 1.967000e+05 2.044000e+05 2.039667e+05 2.083333e+05 2.109000e+05 2.148000e+05 2.174000e+05 2.235667e+05 2.234333e+05
South Carolina Wedgefield NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 8.096667e+04 8.013333e+04 8.200000e+04 8.130000e+04 7.913333e+04 8.080000e+04 8.110000e+04 8.050000e+04 8.340000e+04 8.630000e+04
Michigan Williamston 1.591667e+05 1.613000e+05 1.643000e+05 1.662000e+05 1.664333e+05 1.686333e+05 1.716667e+05 1.750333e+05 1.786667e+05 1.793333e+05 ... 1.599000e+05 1.630667e+05 1.656667e+05 1.686333e+05 1.704000e+05 1.725000e+05 1.739000e+05 1.753667e+05 1.773000e+05 1.809000e+05
Arkansas Decatur 6.360000e+04 6.440000e+04 6.566667e+04 6.673333e+04 6.720000e+04 6.770000e+04 6.650000e+04 6.540000e+04 6.460000e+04 6.490000e+04 ... 8.696667e+04 8.943333e+04 9.106667e+04 9.170000e+04 9.313333e+04 9.486667e+04 9.683333e+04 9.550000e+04 9.420000e+04 9.466667e+04
Indiana Edgewood 9.170000e+04 9.186667e+04 9.293333e+04 9.490000e+04 9.893333e+04 1.000667e+05 1.008333e+05 1.010000e+05 1.021667e+05 1.017667e+05 ... 9.883333e+04 9.780000e+04 9.753333e+04 9.796667e+04 9.643333e+04 9.866667e+04 1.011000e+05 1.025333e+05 1.021667e+05 1.023667e+05
Tennessee Palmyra NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.218667e+05 1.283000e+05 1.247333e+05 1.209000e+05 1.210000e+05 1.200000e+05 1.243667e+05 1.243333e+05 1.250333e+05 1.260667e+05
Maryland Saint Inigoes 1.480667e+05 1.476000e+05 1.572333e+05 1.633667e+05 1.642333e+05 1.682000e+05 1.665000e+05 1.653333e+05 1.673000e+05 1.688000e+05 ... 2.932333e+05 2.956000e+05 2.878667e+05 2.822667e+05 2.799333e+05 2.778667e+05 2.767000e+05 2.787000e+05 2.814333e+05 2.818667e+05
Indiana Marysville NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ... 1.164333e+05 1.127333e+05 1.093333e+05 1.063667e+05 1.082000e+05 1.122333e+05 1.225000e+05 1.243333e+05 1.251333e+05 1.316000e+05
Missouri Bois D Arc 1.078000e+05 1.070000e+05 1.071000e+05 1.081000e+05 1.107000e+05 1.136667e+05 1.126333e+05 1.127333e+05 1.130667e+05 1.154333e+05 ... 1.395333e+05 1.394000e+05 1.456333e+05 1.452333e+05 1.419333e+05 1.404667e+05 1.451333e+05 1.451333e+05 1.417333e+05 1.434333e+05
Virginia Henrico 1.285667e+05 1.307667e+05 1.322667e+05 1.332667e+05 1.352333e+05 1.367333e+05 1.386000e+05 1.413333e+05 1.435333e+05 1.461333e+05 ... 1.998000e+05 2.035000e+05 2.061000e+05 2.085000e+05 2.123333e+05 2.132667e+05 2.146000e+05 2.143333e+05 2.156667e+05 2.215667e+05
New Jersey Diamond Beach 1.739667e+05 1.831000e+05 1.889667e+05 1.931333e+05 1.944000e+05 2.102667e+05 2.302333e+05 2.486667e+05 2.599333e+05 2.656333e+05 ... 3.822333e+05 3.840333e+05 3.872000e+05 3.919667e+05 3.931333e+05 3.988333e+05 4.001667e+05 4.041667e+05 4.011667e+05 3.983000e+05
Pennsylvania Plymptonville 4.706667e+04 4.993333e+04 5.256667e+04 5.413333e+04 5.393333e+04 5.233333e+04 4.926667e+04 5.396667e+04 5.226667e+04 5.006667e+04 ... 4.020000e+04 4.326667e+04 4.836667e+04 4.903333e+04 4.910000e+04 4.996667e+04 5.333333e+04 5.796667e+04 6.016667e+04 6.253333e+04
Highland Township 1.325000e+05 1.321333e+05 1.340333e+05 1.349333e+05 1.382667e+05 1.432000e+05 1.433667e+05 1.427333e+05 1.420333e+05 1.433333e+05 ... 2.248000e+05 2.242667e+05 2.257667e+05 2.272000e+05 2.294667e+05 2.237333e+05 2.200333e+05 2.178667e+05 2.151333e+05 2.204333e+05
Wisconsin Town of Wrightstown 1.017667e+05 1.054000e+05 1.113667e+05 1.148667e+05 1.259667e+05 1.299000e+05 1.299000e+05 1.294333e+05 1.319000e+05 1.342000e+05 ... 1.479667e+05 1.483667e+05 1.474667e+05 1.488667e+05 1.508333e+05 1.526000e+05 1.532333e+05 1.557333e+05 1.594667e+05 1.641667e+05
New York Urbana 7.920000e+04 8.166667e+04 9.170000e+04 9.836667e+04 9.486667e+04 9.853333e+04 1.029667e+05 9.803333e+04 9.396667e+04 9.460000e+04 ... 1.264333e+05 1.257667e+05 1.286667e+05 1.310667e+05 1.323333e+05 1.311667e+05 1.291000e+05 1.338000e+05 1.375667e+05 1.457667e+05
Wisconsin New Denmark 1.145667e+05 1.192667e+05 1.260667e+05 1.319667e+05 1.438000e+05 1.469667e+05 1.483667e+05 1.491667e+05 1.531333e+05 1.567333e+05 ... 1.684000e+05 1.749000e+05 1.795000e+05 1.827333e+05 1.851667e+05 1.845333e+05 1.846667e+05 1.855333e+05 1.901000e+05 1.964000e+05
California Angels 1.510000e+05 1.559000e+05 1.581000e+05 1.674667e+05 1.768333e+05 1.837667e+05 1.902333e+05 1.845667e+05 1.840333e+05 1.861333e+05 ... 2.146333e+05 2.180000e+05 2.228333e+05 2.301667e+05 2.287000e+05 2.400667e+05 2.532000e+05 2.546667e+05 2.592000e+05 2.644000e+05
New Jersey Lebanon Borough 1.658000e+05 1.698333e+05 1.732667e+05 1.772333e+05 1.803333e+05 1.838000e+05 1.882667e+05 1.918667e+05 1.933667e+05 2.008000e+05 ... 2.330667e+05 2.320333e+05 2.294000e+05 2.320667e+05 2.316333e+05 2.320333e+05 2.334667e+05 2.375000e+05 2.411000e+05 2.394333e+05

10927 rows × 67 columns


In [219]:
def run_ttest():
    '''First creates new data showing the decline or growth of housing prices
    between the recession start and the recession bottom. Then runs a ttest
    comparing the university town values to the non-university towns values, 
    return whether the alternative hypothesis (that the two groups are the same)
    is true or not as well as the p-value of the confidence. 
    
    Return the tuple (different, p, better) where different=True if the t-test is
    True at a p<0.01 (we reject the null hypothesis), or different=False if 
    otherwise (we cannot reject the null hypothesis). The variable p should
    be equal to the exact p value returned from scipy.stats.ttest_ind(). The
    value for better should be either "university town" or "non-university town"
    depending on which has a lower mean price ratio (which is equivilent to a
    reduced market loss).'''
    
    df = convert_housing_data_to_quarters()
    rec_start = get_recession_start()
    rec_bottom = get_recession_bottom()
    df = df.loc[:, rec_start:rec_bottom]
    df.reset_index(inplace=1)
    
    # price ratio
    def price_ratio(row):
        return (row[rec_start] - row[rec_bottom])/row[rec_start]
    
    df['ratio'] = df.apply(price_ratio, axis=1)
    
    # check if university towns
    df_uni_town = get_list_of_university_towns()['RegionName']
    df['uni_town'] = df['RegionName'].isin(df_uni_town.values)
    df_uni = df[df['uni_town'].values]
    df_not_uni = df[df['uni_town'] == False]
    
    # ttest
    uni = df_uni['ratio'].dropna()
    not_uni = df_not_uni['ratio'].dropna()
    
    def better():
        if uni.mean() < not_uni.mean():
            return 'university town'
        else:
            return 'non-university town'

    
    pval = ttest_ind(uni, not_uni).pvalue
    
    different = pval < 0.01
    
    return (different, pval, better())
run_ttest()


Out[219]:
(True, 0.00064350059036584158, 'university town')

In [ ]: